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DEVELOPING  COLLECTIVE  TRAINING  FOR  SMALL  UNMANNED  AERIAL  SYSTEMS 
EMPLOYMENT:  WORKLOAD,  AND  PERFORMANCE  WITH  MULTIPLE  SYSTEMS 

EXECUTIVE  SUMMARY 


Research  Requirement: 


Over  the  last  several  years,  unmanned  aerial  systems  (UASs)  have  been  increasingly 
employed  for  reconnaissance  and  surveillance  by  the  U.  S.  Army.  UASs  have  been  assigned  at 
different  echelons,  typically  with  size  (and  capabilities)  decreasing  with  echelon.  Moreover, 
research  on  developing  ever  smaller  and  easier  to  use  UASs  continues  (e.g.,  the  Defense 
Advanced  Research  Project  Agency  is  now  funding  development  of  nano-UASs).  However, 
assigning  SUASs  to  units  without  dedicated  staffs  or  command  and  control  networks  (such  as  at 
company  and  below)  may  raise  a  number  of  coordination  and  communication  issues.  Currently 
deployed  SUASs  do  not  typically  include  a  networked  means  of  sharing  the  visual  data  they  can 
collect.  Real-time  communication  about  sensor  observations  mostly  relies  on  verbal  information 
sharing  over  FM  radio  nets  and  may  depend  on  the  aid  of  intermediaries  between  the  SUAS 
operator  and  the  ultimate  consumer  of  the  information.  In  addition,  SUAS  employment  requires 
coordination  with  manned  air  assets  to  avoid  collisions  and  with  adjacent  units  with  unmanned 
systems  to  avoid  radio  frequency  conflicts.  Thus,  the  training  requirements  that  will  be  incurred 
by  assigning  SUASs  to  company  and  below  will  go  beyond  operator  and  maintenance  training.  It 
will  require  training  for  leaders  in  mission  planning  and  supervision  to  allow  for  the  coordinated 
employment  of  the  SUAS  in  the  context  of  the  unit’s  larger  overall  mission  (Durlach,  2007). 
Little  attention  has  been  paid  thus  far  to  this  type  of  collective  training  for  the  employment  of 
SUAS.  The  research  requirement  is  to  determine  where  problems  may  arise  in  the  collective 
employment  of  SUAS  and  how  these  can  be  addressed,  by  providing  dedicated  training  on  how 
to  avoid  these  problems,  technological  support  to  help  overcome  these  problems,  or  both.  The  U. 
S.  Army  Unmanned  Aircraft  Systems  Roadmap  (2009)  predicts  that  FY  2012  will  require 
training  for  over  2,100  UAS  operators,  maintainers,  and  leaders.  The  current  project  created  a 
simulation  environment  as  a  mechanism  for  providing  research  on  the  collective  training  needs 
that  will  arise  as  lower  and  lower  echelons  are  assigned  SUASs. 


Procedure: 

To  meet  this  need,  a  research  simulation  environment  was  developed  to  investigate 
training  issues  concerning  employment  of  SUAS  at  company  and  below.  The  research 
environment  enables  simulated  platoon  missions  in  a  virtual  environment  in  which  mounted  or 
dismounted  Soldiers  control  avatars.  A  person  designated  as  the  SUAS  operator  has  the  ability  to 
operate  a  SUAS  in  the  virtual  environment.  Another  person  designated  as  the  commander  has  a 
command  and  control  tool  (C2node),  which  allows  communication  and  coordination  with  the 
SUAS  operator  (e.g.,  text  messages,  mission  plans,  sensor  imagery).  Both  the  operator’s  control 
station  and  the  C2node  are  configurable,  to  allow  investigation  of  how  different  features  (e.g., 
providing  the  C2node  with  streaming  video  vs.  not)  affect  mission  synchronization  and 
maintenance  of  a  common  operational  picture.  Besides  human-controlled  avatars,  the 
environment  also  allows  for  artificially  intelligent  non-player  characters  (NPCs),  whose  behavior 
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can  be  scripted  via  a  system  of  relatively-user  friendly  menus,  prior  to  a  scenario  exercise. 
Experimenters  can  also  take  manual  control  of  these  NPCs  during  an  exercise.  This  report 
describes  the  capabilities  of  the  environment  and  an  initial  experiment  providing  evidence  that 
users  can  operate  the  C2node  and  an  avatar  at  the  same  time. 

After  putting  these  capabilities  into  place,  an  initial  experiment  was  conducted  to 
determine  the  feasibility  of  the  commander  being  able  to  operate  both  the  C2node  and  their 
avatar  simultaneously.  Because  research  participants  were  from  a  civilian  population,  the 
scenario  used  during  this  experiment  was  a  non-military  search  and  identification  mission.  This 
task  was  created  to  incorporate  activities  that  would  be  elements  of  more  realistic  maneuver 
platoon  missions.  It  required  each  participant  to  identify  objects  in  pictures  sent  to  the  C2node 
and  simultaneously  to  identify  specific  avatars  within  the  virtual  environment.  Workload  was 
varied  over  trials  by  additional  tasks  being  added  to  both  the  C2node  and  the  virtual 
environment.  Participants  performed  over  four  trials  which  consisted  of  high  and  low  workload 
combinations  within  both  C2node  and  the  game  environment.  Specifically,  the  four  trials 
consisted  of  the  baseline  tasks  (search  and  identify  tasks),  the  baseline  plus  an  additional  task  in 
C2node,  baseline  plus  an  additional  task  in  the  game  environment,  and  baseline  plus  additional 
tasks  in  both  C2node  and  the  game  environment. 

Findings: 

Participants  were  able  to  operate  the  two  system  components,  with  performance 
improving  over  trials.  Task  performance  was  affected  by  both  video  game  use  and  spatial  ability, 
with  higher  spatial  ability  and  more  video  game  use  associated  with  higher  performance. 
Performance  and  perceived  workload  were  significantly  affected  by  the  workload  manipulation, 
but  the  effects  were  small  compared  with  the  beneficial  effects  of  practice  (i.e.,  participant 
performance  improved  over  practice  trials),  which  bodes  well  for  the  use  of  this  environment  to 
investigate  more  tactically  relevant  mission  scenarios.  Additionally,  based  on  usability  questions, 
the  overall  reaction  to  and  perceived  usability  of  the  system  was  favorable. 


Utilization  and  Dissemination  of  Findings: 

The  procedures  the  participants  were  asked  to  perform  in  this  experiment  were  different 
than  those  of  a  platoon  leader  in  the  field.  Nevertheless,  the  experiment  demonstrated  that  users 
can  quickly  learn  to  use  the  C2node  and  the  game  environment  simultaneously.  Furthermore, 
despite  perceptions  of  workload  and  differences  in  performance  based  on  workload  conditions, 
participants  still  improved  overall  by  the  fourth  trial.  This  indicated  that  the  environment  has  the 
potential  to  support  investigation  of  more  mission-relevant  behaviors.  This  work  has  been 
presented  at  the  conference  MODSIM  WORFD  2010.  In  addition,  the  software  has  been  made 
available  to  contractors  (i.e.,  SA  Technologies,  Design  Interactive,  Alion,  Inc.)  for  research 
purposes. 
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DEVELOPING  COLLECTIVE  TRAINING  FOR  SMALL  UNMANNED  AERIAL 
SYSTEMS  EMPLOYMENT:  WORKLOAD  AND  PERFORMANCE  WITH  MULTIPLE 
SYSTEMS 


Introduction 

Over  the  last  several  years,  unmanned  aerial  systems  (UASs)  have  been  increasingly 
employed  for  reconnaissance  and  surveillance  by  the  U.  S.  Army,  which  has  over  4,000  UASs  of 
various  sizes  and  capabilities  at  its  disposal  (U.S.  Army  Roadmap  for  UAS  2010  -  2035,  2009). 
UASs  have  been  assigned  at  different  echelons,  typically  with  size  (and  capabilities)  decreasing 
with  echelon.  Moreover,  research  on  developing  ever  smaller  and  easier  to  use  UASs  continues 
(e.g.,  the  Defense  Advanced  Research  Project  Agency  is  now  funding  development  of  nano- 
UASs).  The  U.S.  Army  Roadmap  for  UAS  2010  -  2035  (2009)  predicts  that  deploying  SUAS  at 
company  and  platoon  levels  would  increase  UAS -related  training  requirements  by  800%. 

Within  the  SUAS  operational  environment,  there  are  a  number  of  factors  that  create 
complexity  and,  consequently,  bring  about  training  requirements  over  and  above  training  a 
Soldier  to  operate  the  system  (Durlach  2007).  The  effective  utilization  of  SUAS  requires 
coordination  between  the  operator  and  others.  This  coordination  includes  consideration  of 
additional  assets  (e.g.,  other  military  air  elements),  an  understanding  of  commander’s  intent  and 
planning,  preparation  and  coordination  to  carry  out  that  intent.  Little  attention  has  been  paid  thus 
far  to  the  collective  aspects  of  training  for  employment  of  SUAS.  Research  is  required  to 
determine  where  unanticipated  issues  may  arise  in  the  collective  employment  of  SUAS,  and  how 
these  can  be  addressed,  through  dedicated  training  on  how  to  avoid  these  issues,  and 
technological  support  to  help  overcome  them.  The  current  project  created  a  simulation 
environment  as  a  mechanism  for  providing  research  on  the  collective  training  issues  that  will 
arise  as  lower  and  lower  echelons  are  assigned  SUASs. 

Future  training  must  facilitate  learning  of  skills  necessary  for  the  operation  of  SUAS, 
competencies  unique  to  leaders  with  SUAS  assets  under  their  control,  and  leader  and  operator 
coordination  and  information  sharing.  For  example,  operators  must  acquire  skills  related  to 
terrain  analysis,  monitoring  and  target  identification,  while  leaders  require  skills  related  to 
delegation  and  mission  planning.  Leaders  must  assess  situational  factors  that  may  change  minute 
to  minute,  and  promote  overall  mission  goals  through  the  use  of  the  SUAS  (Pavlas  et  al.,  2009). 
On  top  of  these  individual  requirements,  operators  and  leaders  must  also  learn  to  coordinate  with 
each  other,  share  appropriate  information,  and  communicate  effectively.  While  the  individual 
and  collective  elements  are  difficult  enough  to  train,  the  nature  of  current  SUAS  capabilities  (or 
lack  thereof)  does  not  help.  For  example,  the  operator  and  the  leader  are  almost  always 
distributed  with  no  technical  support  to  create  a  common  operating  picture.  The  operator  is  often 
required  to  relate  an  interpretation  of  the  visual  information  received  from  the  SUAS  feed  to  the 
leader  verbally,  over  a  non-dedicated  radio  net.  The  leader  must  then  make  decisions  based  on 
the  operator’s  interpretation.  Additionally,  because  leaders  have  so  many  other  duties,  they 
usually  have  little  if  any  training  or  expert  knowledge  of  the  system  capability  or  the  challenges, 
both  environmental  and  operational,  which  may  influence  system  performance. 
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Further  complicating  the  interaction  and  coordination,  leaders  and  operators  often  do  not 
have  adequate  insight  into  what  the  other  knows  and  needs  to  know.  For  example,  leaders  must 
consider  other  variables  that  the  operator  may  have  no  knowledge  of  (e.g.,  Soldier  ground 
movement  within  an  adjacent  area  of  operation)  and  leaders,  in  turn,  may  not  understand  all  of 
the  constraints  on  operating  the  SUAS.  As  a  result,  this  coordination  requires  a  great  deal  of 
communication,  cooperation,  and  information  sharing  that  goes  beyond  each  individual’s  task 
skills  or  knowledge  (De  Visser,  et  al.,  2008;  LeGoullon,  et  al.,  2010). 

Our  simulation  environment  for  investigating  issues  for  this  domain  allows  for 
conducting  a  near  platoon-sized  exercise  in  a  virtual  world.  In  addition  to  this  world,  the  SUAS 
operator  has  a  simulated  control  station  for  the  aerial  vehicle,  and  the  nominal  leader  has  a 
command  and  control  station.  Both  of  these  stations  are  configurable,  such  that  one  could  test  the 
effectiveness  of  different  (simulated)  technical  solutions  to  ameliorate  coordination  problems,  by 
manipulating  both  the  information  presented  at  each  station  and  the  information  that  can  be 
passed  between  them.  The  resulting  simulation  and  an  initial  investigation  of  its  usability  is 
described  below.  While  future  research  will  ideally  delve  into  more  complex  issues  described  in 
the  preceding  paragraphs,  this  investigation  was  intended  to  test  the  practicality  and  effectiveness 
of  using  such  a  complex  environment  (e.g.,  multiple  interfaces  and  functionality  requiring 
multitasking).  We  first  describe  the  components  of  research  simulation,  and  then  describe  the 
details  of  our  initial  experiment. 


Research  Simulation  Environment 
Operator  Training  Simulation:  Initial  Research 

Through  collaboration  with  the  Institute  for  Simulation  and  Training  (1ST)  at  the 
University  of  Central  Florida  and,  later,  Research  Network  Inc.  (RNI),  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  began  the  investigation  of  simulation- 
based  training  in  UAS  with  the  development  of  a  research  platform  that  would  enable  the  testing 
and  development  of  simulation-based  training  exercises  and  performance  measures.  The  goal  of 
this  research  was  to  establish  appropriate  training  and  measurement  standards  for  SUAS 
operators  and  commanders.  The  first  step  in  this  process  was  to  create  a  simulated  operator 
control  unit  (OCU)  for  SUAS  operators,  which  was  completed  in  prior  research  (Billings  & 
Durlach,  2008;  Durlach,  Neumann,  &  Billings,  2008). 

Simulated  SUAS 

The  Simulated  SUAS  (SSUAS)  is  loosely  based  on  a  prototype  Micro-Unmanned  Aerial 
Vehicle  (t-MAV)  developed  under  the  Defense  Analysis  Research  Project  Agency’s  MAV 
Technology  Demonstration.  The  SSUAS  is  similar  to  the  t-MAV  (see  Figure  1),  incorporating  a 
ducted-fan  vertical  lift  design  with  the  ability  to  hover,  rotate,  and  travel  at  an  airspeed  of  up  to 
six  knots  under  manual  control  and  over  25  knots  under  waypoint  navigation.  Flight  models  were 
developed  that  allow  the  simulated  SUAS  to  mimic  some  of  the  inertial  properties  of  the  t-MAV. 
The  simulated  SUAS  has  two  cameras,  one  forward  facing  and  one  downward  facing.  The 
simulated  SUAS  is  configurable  so  that  certain  aspects  of  the  simulated  SUAS  can  be 
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manipulated  to  investigate  different  aspects  of  performance.  For  example,  the  cameras  can  be 
configured  to  zoom,  pan,  or  remain  fixed. 


Figure  1.  Simulated  SUAS  in  simulated  environment  used  with  the  simulated  OCU  and 
subsequent  research  platform  expansion. 


Operator  Control  Unit 

The  simulated  operator  control  unit  (SimOCU)  allows  a  human  to  control  the  SSUAS. 
The  SimOCU  display  includes  an  altimeter,  camera  views  with  a  heading  tape,  a  flight  controls, 
options  for  developing  flight  paths,  and  an  overhead  map  showing  the  location  of  the  simulated 
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SUAS  within  a  simulated  military  operations  on  urban  terrain  (MOUT)or  urban  operations  (UO) 
environment  (See  Figure  2).  Furthermore,  the  icons  on  a  tool  bar  along  the  top  of  the  display 
allow  for  operators  to  program  automated  flight  paths,  preset  waypoints,  and  launch  or  interrupt 
the  SSUAS  along  a  predetermined,  automated  path.  Like  the  simulated  SUAS,  the  SimOCU  is 
reconfigurable  as  well,  enabling  for  further  investigation  into  operator  performance.  For 
example,  the  SimOCU  can  be  configured  to  use  a  computer  mouse  or  a  game  controller  to 
operate  the  simulated  SUAS.  The  SimOCU  is  written  in  Linux  using  freely  available  software 
(Open  Scene  Graph  for  rendering  and  OpenAL  for  audio)  and  requires  no  additional  licenses. 
The  SSUAS  and  a  base  station  (i.e.,  a  simulated  military  vehicle  where  the  operator  is  supposed 
to  be  within  the  simulated  environment  during  operation  of  the  SUAS)  are  transmitted  using  the 
Distributed  Interactive  Simulation  (DIS)  protocol  so  that  both  can  be  displayed  in  other  systems, 
enabling  networking  and  sharing  of  information  or  output  by  multiple  systems.  Any  modem  PC 
and  video  card  can  satisfactorily  run  the  SimOCU. 

In  addition  to  the  SimOCU,  synthetic  environments  for  the  simulated  SUAS  to  operate 
within  were  created.  Specifically,  environments  were  created  based  on  two  Army  MOUT  or  UO 
training  sites,  simulating  small  towns  with  different  features  used  in  live  training  exercises. 

While  the  overhead  map  feature  requires  an  additional  image  file,  any  OpenFlight  database  can 
be  loaded  and  a  map  or  an  aerial  view  can  be  used.  For  the  display  and  routing  of  various  types 
of  routes  and  dismounted  personnel,  other  entities  can  be  imported  through  DIS.  For  example, 
we  successfully  interoperated  with  OneSAF  Testbed  Baseline  v2.5  in  order  to  populate  the 
environment  with  entities. 

Prior  Research  Findings 

A  full  account  of  the  research  using  the  SimOCU  can  be  found  in  the  literature  (Billings 
&  Durlach,  2008;  in  press;  Durlach,  Neumann,  &  Billings,  2008).  These  studies  concentrated  on 
two  types  of  missions:  flight  skills  missions,  and  reconnaissance  missions.  Over  three  studies, 
participants  were  given  initial  introductory  training  on  how  to  use  the  SimOCU  and  then  were 
asked  to  perform  a  series  of  missions  involving  either  navigation  of  the  environment  or  detecting 
and  photographing  enemy  targets.  In  all  missions,  participants  controlled  the  SSUAS  through  the 
SimOCU.  Different  SimOCU  configurations  were  used  based  on  the  goals  of  the  individual 
studies,  and  the  sensitivity  of  various  measures  (e.g.,  number  of  collisions,  number  of  targets, 
time  to  complete  mission)  were  examined  based  on  performance  effects  of  these  configurations. 
The  results  of  these  studies  using  the  SimOCU  suggested  the  most  sensitive  performance 
measures  for  training  operators  were  temporal  measures  (i.e.,  time  to  complete  mission).  This 
implies  that  standards-based  training  should  incorporate  temporal  measures  with  other  crucial 
but  less  sensitive  performance  indicators  (e.g.,  number  of  targets  detected). 
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Figure  2.  Example  of  SimOCU  interface. 
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Team-Level  Mission  Simulation 


While  the  importance  of  training  operators  of  small  unmanned  aerial  systems  (SUASs) 
has  been  embraced  by  military  and  training  professionals,  less  attention  has  been  devoted  to  how 
to  train  the  effective  and  efficient  coordination  of  SUASs  within  teams.  At  a  minimum,  the 
operator  will  need  to  coordinate  with  either  a  robotics  noncommissioned  officer  (NCO)  or  team 
leader.  These  leaders  will  require  training  on  how  to  plan  missions  that  integrate  SUAS,  how  to 
take  into  account  air  space  and  frequency  management,  and  how  to  communicate  commander’s 
intent  regarding  the  SUAS  asset.  To  that  end,  the  leader,  operator,  and  other  unit  Soldiers  must 
be  able  to  communicate,  coordinate,  and  share  a  common  picture  of  the  task,  equipment,  and 
other  team  or  squad  members.  To  address  these  emerging  needs,  we  added  new  components  to 
the  original  simulation  environment:  a  command-and-control  node  (C2Node),  created  by  1ST, 
and  a  virtual  environment  for  dismounted  infantry  missions  with  the  simulated  SUAS  (GDIS  or 
Game  Distributed  Interactive  System),  developed  by  Research  Network  Inc  (RNI).  The  original 
components  were  also  modified  to  accommodate  communication  between  the  C2Node  and  the 
SimOCU. 

Specifically,  following  the  initial  operator  research,  the  research  platform  was  expanded 
to  include  three  interacting  components  referred  to  in  the  introduction:  1)  GDIS:  a  virtual 
immersive  environment  that  replicates  one  of  the  synthetic  MOUT  sites  and  can  be  populated 
with  human-controlled  avatars  and  semi-intelligent  computer  generated  forces  (nonplayer 
characters  or  NPCs),  2)  C2Node:  a  command  and  control  node  enabling  communications 
between  the  commander  and  SSUAS  operator,  and  3)  the  modified  SimOCU.  Each  platform 
element  is  run  on  a  separate  computer,  which  means  that  a  minimum  of  three  networked 
computers  are  needed  to  successfully  run  all  simulation  components  simultaneously.  Each  of  the 
components  is  briefly  described  next. 

C2Node 

The  C2Node  simulates  elements  of  command  and  control  systems  such  as  Force  XXI 
Battle  Command  Brigade  and  Below  (FBCB2).  This  component  provides  a  way  for  a  leader  to 
practice  SUAS  information  management  and  decision  making  in  a  simulated  environment.  The 
C2Node  is  almost  fully  reconfigurable  and  consists  of  a  number  of  optional  features  including  an 
interactive  map,  blue  force  tracking,  annotation,  a  window  where  streaming  video  or  pictures  are 
received,  text  windows  for  sending  and  receiving  information,  and  menus  for  mission  planning 
(see  Figure  3).  Options  available  to  reconfigure  include  no  imagery,  still  pictures  only,  or 
streaming  video;  text  versus  voice  communication;  and  options  for  the  map  to  display  blue  forces 
(i.e.,  friendly  forces)  or  not.  In  fact,  because  this  system  is  reconfigurable  across  several 
variables,  one  of  the  main  benefits  of  this  system  is  that  it  allows  the  researcher  to  limit  or 
expand  the  information  available  to  the  leader  in  order  to  assess  the  impact  on  mission 
coordination.  For  example,  the  impact  on  mission  performance  of  the  leader  having  no  imagery, 
still  images,  or  streaming  video  could  be  investigated.  C2Node  features  also  allow  the  user  to 
plan  missions  with  routes,  no  fly  zones,  and  flagged  entities.  Again,  dependent  on  selected 
configuration,  it  may  be  possible  to  send  these  plans  to  the  SimOCU  operator.  For  research 
purposes,  the  C2Node  logs  user  interaction  data,  recording  all  information  exchanged  between 
the  SimOCU  and  the  C2Node. 
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Modified  SimOCU 


The  SimOCU  was  upgraded  with  additional  features  to  allow  communication  with  the 
C2Node  operator,  including  the  exchange  of  artifacts  (e.g.,  pictures,  routes)  with  the  C2Node. 

For  example,  the  SimOCU  operator  can  send  and  receive  text  messages,  pictures,  and  mission 
plans  with  the  C2Node.  The  SimOCU  is  still  reconfigurable  and,  like  the  C2Node,  logs  data  in 
the  form  of  a  text  file. 

GDIS  Virtual  Environment 

GDIS  is  an  immersive  virtual  environment,  similar  to  a  first-person  shooter  video  game. 
GDIS  was  tailored  by  RNI  to  interact  with  the  C2Node  and  SimOCU,  but  it  can  also  function 
independently  (Barnett  &  Taylor,  2010).  GDIS  allows  for  the  expansion  of  the  dyad  (i.e.,  leader 
and  OCU  operator)  to  include  other  characters,  including  Soldiers,  enemy,  and  bystanders  (See 
Figure  5).  When  networked  with  the  C2Node  and  the  SimOCU,  a  player  using  GDIS  is  able  to 
observe  and  be  observed  by  the  SUAS.  GDIS  includes  functionality  that  allows  multiple 
distributed  human  players  to  control  avatars,  which  can  move,  shoot,  emote,  and  communicate 
with  other  players.  The  GDIS  system  contains  substantial  artificial  intelligence  (AI)  elements  as 
well.  The  AI  in  particular  allows  for  multi-person  scenarios  with  fewer  human  operators.  This 
allows  for  platoon-sized  missions  and  the  integration  of  the  employment  of  the  SSUAS  into 
those  missions.  Mission  action  is  supported  by  experimenters  and  the  Al-controlled  NPCs.  For 
example,  an  operator-in-training  can  use  the  SimOCU  to  fly  a  SUAS  through  the  virtual 
environment  to  detect  IEDs  in  relation  to  friendly  forces  in  the  environment.  When  these  targets 
are  detected,  the  operator  can  then  relay  this  information  to  the  commander  at  the  C2node.  Based 
on  this  information,  the  commander  can  communicate  changes  in  mission  plans  or  routes  to  the 
human  players  in  GDIS  or  can  use  a  menu  system  to  re-route  Al-based  NPCs.  Thus,  the  three 
different  components  of  the  research  platform  can  work  together  to  investigate  various  aspects  of 
team  performance  when  robotic  assets  are  used. 
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Figure  4.  Modified  SimOCU  interface. 
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Authoring  tools  in  GDIS  allow  an  experimenter  to  author  specific  scenarios  as  well  as 
change  aspects  of  scenarios  during  game  play.  Menu-based  authoring  capabilities  are  included  in 
GDIS,  which  allow  for  scenario  generation  functions,  including  mission  planning,  route 
generation,  route  assignments,  insertion  of  objects  and  vehicles,  and  the  ability  to  ‘take  over’  a 
non-player  character  (NPC)  or  switch  perspectives  within  game  play.  For  example,  through  point 
and  click  menus,  scenario  authors  can  create  and  insert  NPCs  with  a  number  of  characteristics, 
including  team  membership,  weaponry,  competency  (i.e.,  novice,  expert  fighter),  and  rules  of 
engagement  (ROEs).  Routes  can  also  be  created  through  simple  menus  using  waypoints.  NPCs 
can  be  assigned  to  routes  and  specific  instructions  can  be  associated  with  any  waypoint,  to 
control  their  behavior.  For  example,  a  waypoint  may  include  the  instruction  for  the  NPC  to  wait 
at  that  point  for  a  specified  amount  of  time  or  until  some  other  specific  event  occurs  (e.g.,  a  team 
member  reaches  some  other  waypoint).  Thus,  NPC  behaviors  can  be  made  contingent  on  other 
events  or  can  be  triggered  by  events  in  the  environment,  such  as  a  nearby  explosion  or  a  decrease 
in  health  status  of  another  NPC  or  team  member.  For  example,  if  an  improvised  explosive  device 
(IED)  explodes  a  NPC  team’s  mission  might  change  from  defensive  to  offensive.  These  if/then 
contingencies  are  specified  through  the  menu-based  authoring  system  in  the  same  manner  as  the 
waypoint-associated  options  (see  Figure  6  for  an  example  of  menu  driven  generation). 

In  addition  to  controlling  individual  NPC  behaviors,  the  menus  also  contain  options  for 
team  level  behaviors,  which  are  designed  to  be  conducted  by  NPCs  in  a  coordinated  way  without 
individually  scripting  NPCs  (e.g.,  building  search).  GDIS  users  also  have  the  option  of  taking 
control  of  NPCs  during  actual  game  play,  meaning  they  can  manually  control  their  behavior  for 
periods  of  time  as  they  see  fit  and  then  return  the  NPC  to  autonomous  mode. 

GDIS  has  been  designed  with  flexibility  and  interoperability  in  mind.  GDIS  SimBridge 
has  been  designed  to  allow  interaction  with  other  simulation  or  game  platforms  and  to  leverage 
new  technologies  easily  as  they  become  available.  Currently,  GDIS  interfaces  with  the  HF2 
engine  Mod  Type  and  the  GameBryo  engine  (Source  Type). 

Integration  of  Research  Platform  Components 

The  simulation  environment  was  designed  to  enable  realistic  missions  (e.g.,  platoon  to 
enter  town  and  occupy  a  building)  in  order  to  identify  team  training  requirements  and  to 
investigate  how  team  coordination  is  affected  by  the  functionality  provided  at  the  C2Node  and 
SimOCU.  This  information  can  then  be  used  to  determine  how  to  design  collective  training,  as 
well  as  the  nature  of  the  tools  that  would  optimally  support  coordination.  Before  such  research 
can  be  initiated,  however,  we  wanted  to  determine  how  much  of  the  functionality,  available 
within  the  expanded  research  platform,  could  be  managed  and  utilized  by  a  participant- 
commander  using  both  the  C2Node  and  GDIS  simultaneously.  That  is,  we  wanted  to  make  sure 
that  a  person  playing  the  role  of  a  platoon  leader  could  handle  the  workload  of  controlling  an 
avatar  in  GDIS  and  operating  the  C2Node  at  the  same  time. 
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Figure  5.  GDIS  environment  with  NPCs  and  the  simulated  SUAS. 
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Figure  6.  GDIS  route  menu. 
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Experiment 


Workload 

There  are  many  real  world  situations  where  individuals  are  required  to  perform  multiple 
tasks  simultaneously;  however,  even  when  systems  intended  to  make  this  easier  are  put  into 
place,  they  can  often  increase  workload,  especially  when  proper  training  is  not  provided.  Multi¬ 
tasking  can  often  lead  to  decrements  in  performance,  feelings  of  being  overwhelmed,  or  neglect 
of  some  of  the  tasks.  This  is  due  in  large  part  to  individual  characteristics  and  the  limited 
capacity  of  working  memory  in  general  (Mayer,  2001).  Thus,  SUASs,  intended  to  provide  teams 
with  additional  information,  may  not  necessarily  result  in  improved  performance  if  managing  the 
employment  of  the  system  adds  too  much  to  workload. 

An  increase  in  workload  in  general  has  been  found  to  change  the  way  that  individuals  and 
team  members  make  decisions.  For  example,  Drabek  and  Haas  (1969)  found  that  team  members 
switched  from  semi-autonomous  decision  making  to  more  consultative  decision-making  as 
workload  increased.  In  this  experiment,  team  members  showed  a  need  for  confirmation  or  input 
from  others  when  making  decisions  under  high  stress,  whereas,  they  were  able  to  make  more 
independent  decisions  when  under  low  stress.  Furthermore,  results  showed  that  communication 
changed  under  periods  of  high  stress.  Requests  for  information  by  lower  ranking  team  members 
were  directed  primarily  to  higher  ranking  team  members  as  workload  increased.  Team  members 
also  sought  to  communicate  more,  including  seeking  alternative  ways  of  overcoming  the 
limitations  of  structured  communication  systems  used  during  lower  rates  of  workload.  More 
recent  research  has  provided  further  support  to  Drabek  and  Haas’  (1969)  findings  such  that 
increased  workload  often  leads  to  tendencies  to  conform  to  the  decisions  of  the  leader  and/or  the 
majority  and  a  centralization  of  power/influence  by  leaders  (De  Grada,  Kruglanski,  Mannetti 
Pierro,  1999;  Kruglanski  &  Webster  1996;  Pierro,  Mannetti,  De  Grada,  Livi,  &  Kruglanski, 
2004).  However,  the  primary  way  that  workload  has  been  found  to  impact  performance  is  from 
errors  that  result  from  user  difficulty  in  gathering  and  selecting  critical  information  and  decision 
making  accuracy  (Kawano,  Fujiie,  Ujita,  Kubota,  Yoshimura,  &  Ohtsuka,  1991). 

There  is  also  a  great  deal  of  research  that  illustrates  the  negative  impacts  of  workload 
with  regards  to  new  technology  and  operators.  For  example,  research  has  shown  that  fewer 
participants  finished  missions  using  unmanned  ground  vehicles  or  performed  effectively  when 
presented  with  the  opportunity  to  use  multiple  sensors  (e.g.,  Rehfeld,  Jentsch,  Curtis,  & 
Fincannon,  2005)  or  UASs  (e.g.,  Dixon,  Wickens,  &  Chang,  2003)  to  gather  more  information 
when  compared  with  participants  who  only  used  a  single  source.  This  supports  the  adage  that 
more  is  not  always  better  when  it  comes  to  the  use  of  new  technology  to  gather,  process,  and 
share  information. 

As  a  result,  our  initial  research  effort  with  the  expanded  platform  focused  on  issues  of 
workload.  The  experiment  was  designed  to  evaluate  the  capacity  of  individuals  to  operate  their 
avatar  in  GDIS  and  use  the  capabilities  of  the  C2Node  at  the  same  time,  with  varying  degrees  of 
imposed  workload.  The  C2Node  has  been  designed  so  that  the  experimenter  can  manipulate  its 
functionality  as  well  as  control  the  amount  of  information  channeled  by  each  function.  We  used 
some  of  these  capabilities  to  determine  the  level  of  C2Node  functionality  and  information  an 
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individual  could  deal  with,  while  still  adequately  conducting  tasks  in  GDIS.  We  measured  both 
perceived  workload  and  performance  during  low  and  high  workload  conditions. 

Usability  &  Utility 

Because  this  was  the  first  research  project  using  the  integrated  simulation,  we  also 
investigated  usability.  A  training  system  can  only  be  effective  if  it  is  relatively  easy  to  use.  The 
trainee  should  be  thinking  about  what  is  happening  in  the  training  scenario  and  how  to 
successfully  complete  it,  not  the  knobology  of  the  user-interface.  Usability  of  training 
simulations  and  research  platforms  must  be  systematically  evaluated  so  that  they  are  both 
functional  and  easy  to  use  (Nielsen,  1994).  The  system  interface,  functionality,  and  artifacts 
should  be  evaluated  to  ensure  that  they  can  be  used  easily,  effectively,  and  satisfactorily  by  users 
performing  specific  tasks  in  specific  environments  (Shackel,  1991).  While  there  are  a  number  of 
available  methodologies  for  evaluating  the  usability  of  a  system,  one  popular  and  easily 
administered  method  of  usability  evaluation  is  the  user  reactions  survey.  To  that  end,  a  user 
survey  was  designed  to  target  user  reactions  and  evaluations  of  user  interactions  with  both  the 
GDIS  and  C2Node  systems.  Participants  were  able  to  give  their  reactions  and  perceptions  of  ease 
of  use  for  both  systems.  These  reactions,  reported  below,  can  inform  future  iterations  of  this 
research  platform,  as  well  as  the  design  of  training  using  the  different  configurations  available. 
To  achieve  this  goal,  reactions  to  both  the  interface  and  the  functionality  of  GDIS  and  the 
C2Node  were  evaluated  to  ensure  that  these  systems  can  be  used  by  the  targeted  population  and 
that  the  required  functions  are  present.  Data  consisted  of  surveys  (e.g.,  written  questions 
regarding  ease  of  use  which  were  rated  on  a  five  point  Likert  scale)  based  on  standard  usability 
questions  presented  in  the  literature  (Nielsen,  1994;  Rossen  &  Carroll,  2002). 


Research  Objectives 

As  stated,  the  main  objective  was  to  evaluate  whether  participants  could  operate  the 
C2Node  and  GDIS  simultaneously.  Participants  performed  four  different  search  and 
identification  tasks,  in  which  task  requirements  were  varied  for  a  total  of  four  practice  trials.  Two 
versions  of  the  GDIS  (low  vs.  high  demand)  and  two  versions  of  the  C2node  task  (low  vs.  high 
demand)  were  crossed,  so  that  each  participant  performed  each  of  four  different  levels  of  task 
demand  over  four  trials. 

Baseline  (low  workload)  tasks  were  developed  for  GDIS  and  the  C2Node,  along  with 
workload  tasks  for  each.  The  baseline  tasks  were  designed  to  utilize  a  portion  of  the 
functionality  of  each  system  that  seemed  likely  to  be  included  in  future  training  and 
experimentation  (e.g.,  search  and  identification  tasks).  The  goals  were  to  determine  1)  if  users 
could  effectively  complete  baseline  tasks  using  both  system  components  simultaneously,  2)  if 
added  workload  kept  them  from  performing  baseline  tasks  effectively  using  both  system 
components  simultaneously,  and  3)  if  users  found  both  systems  usable  and  had  positive  reactions 
overall.  The  mission  scenario  used  was  not  tactical  because  of  the  available  sample  (i.e.,  college 
students);  however  it  did  include  GDIS  and  C2Node  operations  likely  to  be  required  in  more 
militarily  relevant  scenarios. 
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Method 


Participants 

Data  were  collected  using  a  student  population  recruited  from  the  local  university.  The 
goal  sample  size  was  32  for  a  within  group  manipulation  of  both  trial  and  workload  (i.e.,  each 
participant  performed  in  4  trials  and  experienced  4  levels  of  workload).  Although  the  sample  size 
is  relatively  small,  the  within-subjects  design  increases  the  power  because  each  participant 
provides  4  data  points  by  experiencing  each  workload  condition  across  the  four  trials. 
Specifically,  each  workload  condition  had  32  participants  because  each  participant  experienced 
each  level  of  workload  and  each  participant  had  four  trials.  Furthermore,  tests  of  Normality 
specifically  targeting  small  sample  sizes  showed  a  normal  distribution,  supporting  the  inclusion 
of  parametric  testing.  Specifically,  the  Shapiro-Wilk  Test,  which  is  more  appropriate  for  small 
sample  sizes  (<  50  samples),  was  used  to  test  as  our  numerical  means  of  assessing  normality  (i.e., 
with  this  specific  test,  if  the  significance  value  is  greater  than  0.05,  then  the  data  is  normal).  See 
Table  1  for  a  list  of  significance  values.  As  can  be  seen,  the  significance  levels  for  all  four  trials 
are  greater  than  0.05,  indicating  a  normal  distribution. 


Table  1 


Tests  of  Normality 


Trial 

Kolmogorov-Smirnova 

Shapiro-Wilk 

Statistic 

df 

Sig. 

Statistic 

df 

Sig. 

Overall  1 

.289 

8 

.047 

.839 

8 

.074 

Score  2 

.197 

8 

.200 

.908 

8 

.342 

3 

.190 

8 

.200 

.909 

8 

.347 

4 

.215 

8 

.200 

.928 

8 

.494 

a.  Lilliefors  Significance  Correction 

*.  This  is  a  lower  bound  of  the  true  significance. 


No  specific  qualifications  were  required  other  than  those  participants  be  over  18  years  of 
age  and  have  normal  or  corrected  vision.  Participants  received  extra  credit,  with  the  amounts 
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based  on  length  of  their  participation  in  the  research  (e.g.,  full  participation  warranted  two  extra 
credit  points). 


Search  and  Identification  Scenarios 

The  scenarios  required  participants  to  use  both  the  C2Node  and  GDIS  simultaneously. 
Participants  were  given  instructions  that  they  were  in  command  of  US  troops  who  are  part  of  a 
platoon  that  has  recently  liberated  a  small  village  in  an  unnamed  country.  They  are  told  that  the 
Red  Cross  has  sent  medics  to  care  for  refugees  and  that  there  are  Soldiers  in  the  area  providing 
security  as  well  as  questioning  refugees.  However,  with  all  the  different  organizations  entering 
and  leaving  the  area,  there  is  little  information  being  sent  to  headquarters  on  the  number  and 
location  of  Soldiers,  medics,  and  refugees.  Therefore,  the  task  was  to  identify  the  number  and 
location  of  Soldiers,  medics,  and  refugees  within  the  liberated  village.  Also,  due  to  concern  over 
lingering  enemy  forces,  the  participants  were  told  that  they  will  receive  intelligence  (i.e., 
information  or  pictures)  from  a  micro-unmanned  aerial  vehicle  operated  by  a  Soldier  under  their 
command  and  that  they  must  report  the  Intel  (e.g.,  identify  targets  in  pictures)  to  Headquarters. 

Under  this  premise,  participants  were  asked  to  perform  two  basic  tasks,  one  within  GDIS 
and  one  within  C2Node,  simultaneously.  Within  GDIS,  participants  were  asked  to  navigate  their 
avatar  through  buildings  circled  on  maps  (different  buildings  for  each  trial)  and  identify  Soldiers, 
medics,  and  refugees  within  buildings,  logging  these  on  a  paper  form.  Within  the  C2Node, 
participants  were  asked  to  identify  the  presence  or  absence  of  vehicles  or  people  (i.e.,  Humvees, 
none,  unknowns,  and  Soldiers)  for  Headquarters,  based  on  pictures  received  on  the  C2Node 
interface  from  the  SUAS  (which  were  sent  by  the  experimenter  who  was  utilizing  the  SimOCU). 
Participants  were  asked  to  identify  different  items  per  trial,  by  replying  ‘y’  or  ‘n’  in  the  text  box 
within  the  C2Node.  Participant  responses  were  captured  by  the  system  through  the  texting 
capabilities  of  the  C2Node  and  SimOCU.  These  yes  or  no  answers  to  item  detection  served  as 
one  of  the  performance  measures.  The  identification  of  the  number  of  Soldiers,  medics,  and 
refugees  entered  on  the  paper  form  was  another.  These  performance  measures  were  used  with  the 
time  on  task  to  calculate  an  overall  performance  score. 

To  create  the  high  workload  demand  conditions,  an  additional  task  was  added  onto  each 
system.  The  C2Node  extra  task  consisted  of  identifying  the  location  of  the  SUAS  using  the 
interactive  grid  map  and  then  reporting  the  first  four  coordinates  of  its  location  using  the  text 
feature  within  C2Node.  Headquarters  (i.e.,  the  experimenter)  prompted  participants  for  the 
location  of  the  SUAS  through  the  text  feature  of  the  C2Node.  The  GDIS  extra  task  consisted  of 
identifying  the  position  of  a  doctor,  played  by  a  NPC  following  a  pre-scripted  route. 

Headquarters  (i.e.,  the  experimenter)  prompted  the  participants  for  the  location  of  the  doctor 
through  the  text  feature  of  the  C2Node.  To  successfully  perform  this  extra  task,  participants  had 
to  refer  to  two  different  area  maps.  First,  participants  pulled  up  an  interactive  map  in  GDIS  and 
found  the  location  of  the  doctor.  While  the  doctor’s  location  showed  up  on  this  map,  the  building 
numbers  that  needed  to  be  reported  to  Headquarters  did  not.  Therefore,  participants  used  a 
second  paper  map  to  identify  the  specific  building  number  that  the  doctor  was  closest  to.  They 
reported  the  location  using  the  text  function  in  C2Node.  As  a  result,  over  the  4  trials,  participants 
experienced  4  levels  of  workload:  1)  low  GDIS  workload,  low  C2Node  workload  (i.e.,  only  the 
baseline  tasks),  2)  high  GDIS  workload,  low  C2node  workload,  3)  low  GDIS  workload,  high 
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C2Node  workload,  and  4)  high  GDIS  workload,  high  C2Node  workload.  All  communication 
sent  and  received  through  the  C2Node  texting  capability  was  captured  and  time  stamped  for 
analysis. 

Materials 

Paper-and-pencil  or  computer  based  surveys  were  given  pre  and  post-  training. 

Demographics  questionnaire 

The  demographics  questionnaire  consisted  of  questions  regarding  the  age,  gender,  and 
student  status  of  the  individual  (see  Appendix  B).  Also  included  were  a  series  of  questions  to  get 
an  indication  of  the  participant’s  ability  with  computers  and  first-person  shooter  games.  This 
information  was  used  in  the  analyses  to  test  for  the  effects  of  skill  level  workload  and 
performance  within  each  component,  as  well  as  to  compare  different  groups  of  participants  based 
on  computer  and  video  game  use. 

Spatial  ability  measure:  Cube  Comparison  Test.  A  test  was  also  given  to  measure  spatial 
orientation  (i.e.,  the  ability  to  accurately  estimate  changes  in  the  orientation  of  an  object;  Velez, 
Silver,  &  Tremaine,  2005).  The  Cube  Comparison  Test  (CCT)  measures  Spatial  Orientation  by 
presenting  pairs  of  cubes.  Participants  are  asked  to  determine  if  one  cube  represents  a  rotated 
version  of  the  other  cube.  The  test  consists  of  two  3-minute  sections  with  21  questions  each.  The 
higher  the  CCT  score,  the  higher  the  individual’s  spatial  ability. 

Measure  of  workload:  NASA-TLX 

A  measure  of  workload  was  also  used  in  this  research  to  determine  the  perceived 
workload  of  participants  following  each  trial.  Specifically,  participants  were  given  a  computer- 
based  version  of  a  commonly  used  workload  survey  (i.e.,  the  NASA  Task  Load  Index  or  NASA- 
TLX)  to  measure  their  perceptions  of  workload  on  six  dimensions:  mental  demand,  physical 
demand,  temporal  demand,  performance,  effort,  and  frustration.  Hart  and  Staveland’s  (1988) 
NASA-TLX  method  assesses  workload  on  five  7-point  scales.  Increments  of  high,  medium  and 
low  estimates  for  each  point  result  in  21  gradations  on  the  scales.  Participants  are  then  given 
weighted  scores  based  on  pair-wise  comparisons  performed  on  the  six  dimensions.  This  measure 
was  presented  before  training  and  then  again  following  each  trial.  The  participants  were  asked  to 
rate  the  workload  of  the  condition  immediately  preceding  the  administration  of  the  survey.  The 
higher  the  NASA-TLX  score,  the  higher  the  individual’s  perceived  workload. 

Usability  survey 

A  usability  survey  was  given  to  participants  at  the  end  of  the  experiment.  The  surveys 
were  based  on  established  usability  questions  of  reactions  and  interactions  with  systems 
(Neilsen,  1997)  but  were  crafted  specifically  for  each  system  within  this  research  project  based 
on  their  functionality  and  demands  on  users.  The  GDIS  Usability  Survey  (see  Appendix  C)  and 
the  C2Node  Usability  Survey  (see  Appendix  D)  consisted  of  15  questions,  including  a  7-part 
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reaction  question.  Participants  were  asked  to  rank  their  agreement  with  terms  or  statements  based 
on  a  9-point  anchored  Likert  scale. 


Procedure 

Participants  were  asked  to  come  to  the  research  location  and  were  briefed  about  the 
specific  nature  of  the  research,  the  requirements  for  participation  and  tasks,  and  any  risks  that 
may  be  involved  (e.g.,  simulation  sickness).  If  they  agreed  to  participate,  they  then  signed  the 
informed  consent  form  provided.  The  participants  were  then  given  training  sequentially  on  the 
systems  based  on  a  protocol  of  guided  practice.  This  consisted  of  PowerPoint  slides  for  both  the 
GDIS  and  C2Node  systems.  For  GDIS,  these  slides  explained  how  to  move,  identify  items, 
receive  pictures,  and  send  texts.  After  participants  read  through  these  initial  GDIS  slides,  they 
were  required  to  perform  three  tasks  within  GDIS  using  skills  necessary  for  the  experimental 
trials.  They  then  performed  analogous  training  for  the  C2Node.  Most  participants  were  able  to 
perform  adequately;  two  participants  were  excused  because  they  could  not  learn  to  use  GDIS  in 
the  time  allotted. 

Following  training,  participants  were  asked  to  complete  a  demographics  questionnaire 
(Appendix  B),  a  spatial  abilities  test  (the  cube  comparison  test),  and  baseline  measures  of 
perceived  workload  (NASA-TLX).  Once  the  initial  training  and  pre-test  surveys  were  completed, 
participants  were  given  instructions  for  the  experimental  task.  Instructions  and  procedures  were 
scripted  to  ensure  uniform  presentation  across  researchers  (See  Appendix  E).  Participants  were 
then  asked  to  perform  across  4  training  trials.  Each  trial  lasted  between  5  and  15  minutes 
(dependent  on  participant  performance)  and  presented  conditions  of  low  and  high  workload 
described  above.  Performance  measures,  along  with  the  NASA-TLX  measure  of  perceived 
workload,  were  collected  after  each  trial.  Performance  scores  consisted  of  a  composite  score 
based  on  performance  in  the  C2Node  (i.e.,  number  of  items  identified  correctly),  performance  in 
the  GDIS  system  (e.g.,  number  of  Soldiers,  medics,  and  civilians  identified  correctly),  and  the 
time  to  complete  these  tasks  using  a  speed  to  accuracy  equation  (time  allowed  x  number  of 
correct  responses)/time  to  complete;  following  Zandin,  2001).  The  overall  performance  score 
(i.e.,  higher  scores  indicated  better  performance)  and  perceived  workload  scores  (i.e.,  higher 
scores  indicate  higher  perceived  workload)  were  the  repeated  measures. 

Design 


The  experiment  was  a  within  subject  design,  where  each  participant  had  4  trials  and 
experienced  all  levels  of  workload.  The  order  in  which  the  workload  was  presented  was 
randomly  assigned  so  that  the  order  of  presentation  varied.  Specifically,  some  participant  trials 
increased  in  workload  sequentially  (i.e.,  trial  1  had  the  lowest  workload  and  trial  4  had  the 
highest).  Other  participants  trials  decreased  in  workload  (i.e.,  trial  1  had  the  highest  workload 
and  trial  4  had  the  lowest).  While  other  participant  trials  were  varied  randomly  (trial  1  may  have 
had  high  workload  for  the  C2Node  but  not  the  GDIS  task,  while  trial  4  had  high  workload  for 
GDIS,  but  not  for  C2Node).  However,  each  participant  had  4  training  trials  and  experienced  all  4 
levels  of  workload. 
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Results  &  Discussion 


Data  Analysis 

Analyses  of  Covariance  (ANCOVAs)  were  conducted  for  performance  over  the  four 
trials  (one-way  repeated  measure)  as  well  as  for  workload  (GDIS  x  C2Node  workload),  where 
the  dependent  variable  was  the  overall  performance  of  participants  on  the  task  described  above 
and  the  covariates  were  video  game  experience  and  spatial  ability.  An  additional  ANCOVA  was 
performed  for  perceived  workload  scores  based  on  task  loading  (i.e.,  workload).  Post-hoc  tests 
(e.g.,  pairwise  comparisons,  Bonferroni  corrections)  were  also  performed  when  significant 
relationships  were  found  to  identify  the  source  of  significant  differences. 

Demographics  &  Spatial  Ability 

Of  the  32  participants  in  the  final  sample,  35%  of  participants  were  female  (n=  11)  and 
65%  were  male  (n= 21).  A  majority  of  the  participants  used  computers  daily  (n= 29),  with  almost 
everyone  using  computers  five  to  30  hours  per  week  (n= 25).  Seventy-eight  percent  of 
participants  rated  their  computer  skills  as  Novice/Beginner  (n= 27),  with  no  one  claiming  they 
were  an  expert.  A  majority  of  participants  had  previous  remote  control  experience  (n= 25)  and 
owned  a  video  game  system  (n= 25).  While  most  participants  considered  themselves  novice  when 
it  came  to  computer  skills,  only  13%  rated  their  video  game  skills  as  Novice/Beginner  (n=4).  The 
majority  of  participants  rated  their  video  game  skills  as  intermediate  (n=15)  or  expert  (n=  13). 

Demographic  data  and  spatial  ability  were  examined  for  correlations  with  the  dependent 
variables.  Specifically,  it  was  found  that  questions  related  to  video  game  experience  and  spatial 
ability  scores  were  positively  associated  with  overall  performance.  In  particular,  responses  on  a 
question  about  number  of  hours  of  game-play  per  week  were  significantly  correlated  with  overall 
performance,  r  =  .71,  n  =  32,  p  <  .05.  Scores  on  the  Cube  Comparison  Test,  which  tested  spatial 
ability  and  ranged  from  -2  to  36  (p  =  16.19,  SD  =  9.51),  indicated  a  significant  positive 
relationship  between  spatial  ability  and  performance,  r  =  .39,  n  =  32,  p  <  .05.  As  a  result,  both 
hours  spent  playing  video  games  and  the  Cube  Comparison  Test  scores  were  included  as 
covariates  within  the  overall  analyses. 

Performance  on  Multiple  Systems:  Trial  Order 

Figure  7  shows  mean  overall  performance  as  a  function  of  trial  (e.g.,  the  order  of 
training).  A  one-way  repeated  measures  ANCOVA  conducted  on  these  data,  with  video  game 
hours  and  Cube  Comparison  Test  scores  as  covariates,  indicated  a  significant  effect  of  trial,  F{ 3, 
29)  =  6.72,  p  <  .00,  qp"  =  .19.  Post  hoc  comparisons  with  Bonferroni  correction  indicated  that 
trials  one  and  two  failed  to  differ,  and  trials  three  and  four  failed  to  differ;  however,  the  first  and 
second  trial  scores  were  significantly  lower  than  scores  on  the  third  and  fourth  trials  (both  p  < 
.00).  Further  analysis  showed  a  significant  drop  in  time  to  complete  each  trial,  F( 3,  29)  =  8.25,  p 
<  .00,  r|p  =  .47,  with  mean  times  ranging  from  7.54  (trial  1)  to  4.56  (trial  4).  Figure  8  illustrates 
an  increase  in  speed  across  trials  1  to  4  and  Figure  9  shows  that  there  was  no  detectable  decrease 
in  accuracy  across  those  trials. 
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8.00 


Figure  8.  Time  on  task  by  trial. 

*Time  on  task  refers  to  the  time  to  complete  each  trial 

**Trial  number  refers  to  the  order  of  presentation  (i.e.,  trial  1  was  the  first  trial) 
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1.00' 


Figure  9.  Mean  percentage  correct  by  trial. 

*Accuracy  refers  to  percentage  correct 

**Trial  number  refers  to  the  order  of  presentation  (i.e.,  trial  1  was  the  first  trial) 


Performance  on  Multiple  Systems:  Workload  Conditions 

Table  2  shows  mean  performance  scores  for  the  four  task  demand  conditions.  A  2  x  2 
ANCOVA  with  GDIS  workload  (high  vs.  low)  and  C2Node  workload  (high  vs.  low)  as  within- 
subject  factors,  and  Cube  Comparison  Test  and  video  game  hours  as  the  covariates,  revealed  a 
main  effect  of  workload  in  GDIS,  F(l,  29)  =  11.62,p  <  .001,  r\p2  =  .29,  and  C2Node,  F(l,  29)  = 
12.14,  p  <  .001,  r]p2  =  .30.  Based  on  these  differences,  post  hoc  comparisons  were  made  between 
high  and  low  task  loading  for  both  GDIS  and  C2Node.  Results  indicated  a  significant  difference 
between  performance  on  high  and  low  workload  for  GDIS  (Bonferroni  corrected  p  <  .001),  but 
no  difference  between  high  and  low  workload  for  the  C2Node. 
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Table  2 

Mean  Performance  Scores  and  Standard  Deviations  for  Workload  Conditions 


GDIS 

Low 

High 

C2node 

Low 

6.7  (3.1) 

5. 3(2.6) 

High 

6.4(2. 5) 

5. 5(2. 2) 

These  main  effects  were  not  qualified  by  an  interaction  between  GDIS  and  C2Node 
workload,  F(l,  29)  =  0.25 ,p  <  .62.  There  was  a  significant  interaction  between  C2Node 
workload  and  spatial  ability  as  measured  by  the  Cube  Comparison  Test  (CCT)  scores,  F(  1,  29)  = 
16.77,  p  <  .001,  r]p2  =  .37.  Correlations  showed  that  the  CCT  was  highly  correlated  with  low 
C2Node  conditions  but  not  with  high  C2Node  conditions.  In  other  words,  if  there  was  a  high 
level  of  workload  within  the  C2Node,  spatial  ability  failed  to  correlate  with  performance; 
however,  if  workload  within  the  C2node  was  low,  spatial  ability  did  correlate  with  performance. 
Because  the  GDIS  task  had  a  spatial  component  (navigating  around  the  virtual  environment),  but 
the  C2Node  task  did  not,  this  pattern  suggests  that  spatial  ability  had  a  detectable  impact  on 
overall  performance  only  when  the  C2Node  tasks  (nonspatial)  required  relatively  little  effort, 
allowing  them  to  concentrate  on  the  spatially  oriented  GDIS  task.  This  may  have  led  to  an 
advantage  in  overall  performance  for  people  who  had  higher  spatial  ability.  Based  on  this,  scores 
on  each  task  (e.g.,  GDIS  and  C2Node)  were  examined  for  their  relationship  to  spatial  ability.  The 
findings  (See  Table  3)  were  consistent  with  what  we  found  in  the  overall  scores.  That  is,  when 
GDIS  and  C2Node  scores  were  examined  separately,  both  scores  were  correlated  with  CCT  only 
when  there  was  low  workload  in  the  C2Node.  This  is  to  be  expected  since,  while  the  scores  were 
collected  separately,  the  tasks  were  still  being  performed  simultaneously,  so  the  same  effects 
would  be  expected  on  both  scores. 

Performance  on  Multiple  Systems:  Perceived  Workload  on  the  NASA-TLX 

With  regard  to  perceived  workload  as  measured  by  the  NASA-TLX,  a  one-way  repeated 
measures  ANCOVA  using  video  game  experience  and  spatial  ability  as  covariates  was 
conducted,  which  included  NASA-TLX  scores  following  initial  training  and  then  each  trial. 
There  was  a  significant  difference,  F( 4,  28)  =  4.45,  p  <  .05,  r]p2  =  .13.  Post  hoc  comparisons 
revealed  that  the  measure  of  workload  taken  prior  to  the  first  scenario  (i.e.,  training  workload) 
was  significantly  lower  than  all  task  conditions  except  for  LowGDIS,  LowC2Node.  (p ’s  <  .001); 
but  none  of  the  other  comparisons  were  significantly  different. 
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Table  3 


Pearson  Correlation  Matrix  Among  Workload  and  Cube  Comparison  Test  Scores 


Overall  Score 

Low  GDIS, 

High  GDIS, 

Low  GDIS, 

High  GDIS, 

Low  C2Node 

Low  C2Node 

High  C2Node 

High  C2Node 

CCT  Score 

42** 

.59** 

.14 

.13 

GDIS  Scores 

Low  GDIS, 

High  GDIS, 

Low  GDIS, 

High  GDIS, 

Low  C2Node 

Low  C2Node 

High  C2Node 

High  C2Node 

CCT  Score 

47** 

.50** 

.12 

.32 

C2node  Score 

Low  GDIS, 

High  GDIS, 

Low  GDIS, 

High  GDIS, 

Low  C2Node 

Low  C2Node 

High  C2Node 

High  C2Node 

CCT  Score 

.52** 

.44* 

.03 

.19 

**p  <  0.01 

Table  4 

Mean  NASA-TLX  Scores  and  Standard  Deviations  by  Task  Loadings 


Training 

LoGDIS- 

LoC2Node 

HiGDIS- 

LoC2Node 

LoGDIS- 

HiC2Node 

BothWL 

NASA-TLX  Score 

36.3(17.4) 

43.0  (21.0) 

49.1  (19.9) 

45.1 (17.5) 

45.9(19.3) 
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Usability 

The  mean  usability  score  for  all  participants  regarding  the  C2Node  was  7.52  (SD  =  1.01) 
out  of  10  and  ranged  from  4.8  to  8.9.  The  mean  usability  score  for  all  participants  regarding  the 
GDIS  was  7.65  (SD  =  1.05)  out  of  10  and  ranged  from  3.4  to  8.9.  The  lowest  mean  ratings  for 
usability  for  the  C2Node  were  for  questions  about  whether  the  system  was  dull  or  stimulating 
(with  dull  being  the  lower  end  of  the  scale,  with  the  mean  being  5.5).  The  lowest  ratings  for 
usability  for  GDIS  were  for  questions  about  whether  the  system  was  frustrating  or  satisfying 
(with  frustrating  being  the  lower  end  of  the  scale,  with  the  mean  being  6.2).  There  were  no 
significant  effects  for  spatial  ability  or  video  game  use  on  usability  scores  for  either  system. 


Discussion  &  Future  Research 

The  findings  suggest  that  participants  were  quickly  able  to  master  use  of  the  C2node  and 
GDIS  systems  simultaneously.  Overall  performance  improved  rapidly  over  trials,  largely  due  to  a 
decrease  in  time  to  complete,  as  accuracy  was  high  throughout.  While  there  were  differences  in 
performance  based  on  workload,  with  higher  workload  proving  more  difficult,  the  improvement 
gained  from  repetition  was  larger  than  the  effect  of  the  workload  manipulation.  Furthermore, 
while  spatial  ability  and  video  game  experience  did  impact  overall  performance,  spatial  ability 
seemed  to  have  the  biggest  impact  when  C2node  was  the  least  complex,  which  may  mean 
performance  was  more  influenced  by  the  GDIS  tasking.  This  first  investigation  was  relatively 
simple,  requiring  participants  to  walk  around  the  virtual  environment  to  find  and  identify  people 
and  objects,  and  to  process  and  send  text  and  graphics  on  the  C2Node.  Future  research  should 
focus  on  more  tactically  realistic  tasks.  The  present  results  suggest  that  this  simulation 
environment  is  usable  and  can,  therefore,  be  used  to  conduct  research  on  ‘friction  points’  of 
coordination  (i.e.,  types  or  conditions  of  interaction  where  coordination  between  the  leader  (at 
the  C2Node)  and  the  operator  (at  the  SimOCU)  breaks  down  when  bottlenecks  occur,  and  how 
training  can  be  designed  to  help  ease  these  performance  trouble  spots.  To  that  end,  the  software 
has  been  made  available  to  contractors  (i.e.,  SA  Technologies,  Design  Interactive,  Alion,  Inc.) 
for  research  purposes.  The  reconfigurable  nature  of  the  C2Node  and  SimOCU  can  also  be 
capitalized  on  to  investigate  how  features  of  the  technical  tools  provided  can  help  or  hinder 
mission  coordination.  As  a  result  of  the  functionality  developed  and  the  complexity  of  the  SUAS 
domain,  future  plans  for  research  include  team  focused  studies  that  will  examine  the  use  of  both 
adaptive  training  elements  (e.g.,  mastery-based  feedback)  and  synthetic  team  members  to  train 
for  complex  team  interaction  and  coordination. 
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Appendix  A 


List  of  Acronyms 

(Listed  in  alphabetical  order) 


AI  -  Artificial  Intelligence 

ARI  -  Army  Research  Institute 

C2Node  -  Command  and  control  node 

CCT  -  Cube  Comparison  Test 

DIS  -  Distributed  Interactive  Simulation 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and  Below 

GDIS  -  Game  Distributed  Interactive  System 

IED  -  Improvised  Explosive  Device 

1ST  -  Institute  for  Simulation  &  Training 

MAV  (t-MAV)  -  Micro-Unmanned  Aerial  Vehicle 

MOUT  -  Military  Operations  on  Urban  Terrain 

NASA-TLX  -  NASA  Task  Load  Index 

NCO  -  Non-Commissioned  Officer 

NPC  -  Non-Player  Character 

OCU  -  Operational  Control  Unit 

SimOCU  -  Simulated  Operator  Control  Unit 

RNI  -  Research  Network  Inc. 

ROE  -  Rules  of  Engagement 
SUAS  -  Small  Unmanned  Aerial  Systems 
UAS  -  Unmanned  Aerial  Systems 
UO  -  Urban  Operations 
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Appendix  B 


Demographics  Questionnaire 

1.  Year  of  birth: _ 

2.  Gender: 

^  Male  ^  Female 

3.  Have  you  graduated  from  high  school? 
a  Yes  a  No 

4.  Which  hand  do  you  write  with? 

^  Left  ^  Right 

5.  Is  your  vision  in  each  eye  correctable  to  20/20? 

1=1  Yes  1=1  No 

6.  To  your  knowledge,  are  you  color  blind? 

1=1  Yes  1=1  No 

7.  Do  you  own  or  have  access  to  a  computer? 
a  Yes  a  No 

8.  If  yes,  how  often  do  you  use  a  computer? 

^  Daily  ^  Several  times  a  week  ^  Occasionally  ^  Never 

9.  Estimate  how  many  hours  per  week  you  use  a  computer. 

1=1  Never  n  1-5  hrs  1=1  5-10  hrs  n  10-20  hrs  n  20-30  hrs  n  30-40  hrs  n  40  +  hrs 

10.  How  do  you  rate  your  computer  skills? 

^  Novice/Beginner  ^  Intermediate  ^  Expert 
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1 1 .  Do  you  use  the  Internet? 

1=1  Yes  1=1  No 

12.  Do  you  have  any  previous  remote  control  (R/C)  experience  (including  cars,  boats,  etc.)? 

1=1  Yes  1=1  No 

13.  Do  you  own  or  use  a  video  game  system? 

1=1  Yes  1=1  No 

14.  How  would  you  rate  your  video  game  skills? 

^  Novice/Beginner  ^  Intermediate  ^  Expert 

15.  How  often  do  you  play  video  games? 

^  Daily  ^  Several  times  a  week  ^  Occasionally  ^  Never 

16.  Estimate  how  many  hours  per  week  you  play  video  games. 

1=1  Never  n  1-5  hrs  n  5-10  hrs  n  10-20  hrs  n  20-30  hrs  n  30-40  hrs  1=1  40  +  hrs 
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Appendix  C 


GDIS  Usability  Survey 


USABILITY  QUESTIONNAIRE 

Directions:  Please  answer  the  following  questions  as  completely  as  possible  with  regards  to 
computer  application  sou  just  used  (GDIS).  Place  a  circle  around  the  number  that  best  represents 
your  rating  of  your  experience.  and  fill  in  comments  and  or  clarifying  statements,  as  necessary. 
Write  "N/A”  if  feature  is  not  applicable. 

1.  OVERALL  REACTIONS  TO  GDIS  System  C  omments  or  “N/A” 

temble  wonderful 

0123456789  _ 


diffcul:  to  use  easrv  to  ose 

012345678  9 


frustrating  satisfying 

012345678  9 


inadequate  functionality  adequate  functionality 

0  1  2  3  4  5  6  7  8  9 


rigid  flexible 

0123456789 


useless  useful 

0123456789 


dull  stimulating 

0123456789 


INTERACTIONS 

2.  Walking  ?.unning 
difficult 

0  12  3  4 


C  omments  or  “N/A” 


easy 

5  6  7  8  9 


3.  Changing  weapons 

difficult  easy 

0123456789 


4  Identifying  hostages 

difficult  easy 

012345678  9 


fclJCF 
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C  ommenr.  or  “N/A' 


INTERACTION'S  (continued) 

5.  Identifying  enemies 

difficult  easy 

012345678  9 


6.  Identifying  friendly  forces. 

difficult  easy 

012345678  9 


7.  Clearing  buildings 

difficult  easy 

0123456789 


8.  rinding  the  stray  from  building  to  building. 

difficult  easy 

012345678  9 


9.  Using  map 

difficult  easy 

0123456789 


10.  Using  the  keyboard 

difficult  easy 

0123456789 


11.  Using  the  mouse 

difficult  easy 

0123456789 


12.  Insetting  planning  waypoints 

difficult  easy 

0123456789 


13.  Assigning  creating  teams  for  routes. 

difficult  easy 

012345678  9 


14.  The  system  feedback  on  when  buildings  were  cleared  was. 

con&sing  clear 

0  *1  23456789 


15  Both  expenaiced  and  novice  user’s  needs  were  takai  into  consideration. 

never  always 

0123456789 
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C2Node  Usability  Survey 
USABILITY  QUESTIONNAIRE 

Directions:  Please  answer  the  following  questions  as  completely  as  possible  with  regards  to  computer 
application  you  just  used  (C2Node).  Place  a  circle  around  the  number  that  best  represents  your  rating  of 
your  experience.  and  fill  m  comments  and  or  clarifying  statements,  as  necessary.  Write  “N/A”  if  feature 
is  not  applicable. 

OITIUIT  REACTIONS  TO  CINode  System  C  omment.  or  “N/A" 

terrible  wonderful 

0123456789  _ 


difficult  to  use  easy  to  use 

012345678  9 


frustrating  satisfying 

012345678  9 


inadequate  functionality  adequate  functionality 

0123456789 


rigid  flexible 

0123456789 


useless  useful 

0123456789 


dull  stimulating 

0123456789 


INTERACTIONS  Comments  or  “N/A” 

2.  Texting 

difficult  easy 

01234  56789  _ 


3.  Finding  posinons  on  map  grid 

difficult  easy 

0123456789 


4.  Identifying  MAY  position 

difficult  easy 

0123456789 
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Comment:  or  “V.V 


ISTERAC HONS  (conttnned) 

S.  Sending  mission  plan  to  MAV 

difficult  easy 

0123456789 


6.  Using  the  key  board  and  mouse 
difficult 

0  1  2  3  4  5 


easy 

6  7  8  9 


7.  The  functionality  of  the  burtons  for  switching  between  camera  views  was 
confusing  clear 

0  ’1  23456789 


8.  The  functions  of  the  on-screen  manual  control  burtons  (on  the  control  pad)  were 
confusing  dear 

0123456789 


9.  Organization  of  infarmahou  on  the  video  display  screen  was 

confusing  clear 

0123456789 


10.  Current  status  (such  as  Taking  off  Landing.  Grounded)  messages  were 
confusing 


012345678 


clear 

9 


11.  Identifying  entities  in  pictures 

difficult  easy 

0123456789 


12.  Inserting  planning  waypoints. 

difficult  easy 

0123456789 


13.  Insetting  and  formatting  fiags. 

difficult  easy 

012345678  9 


14.  Saving  loading  mission  plans 

difficult  easy 

0123456789 


15.  Using  the  voice  function  to  communicate 

difficult  easy 

01234567S9 
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Experimenter’s  Script 

Experimenter  should  have  practice  maps  loaded. 

Experimenter  should  go  to  confederate  GDIS  computer,  select  create  server,  and  hit  enter. 

Select  Spectate 

Load  Map  (F10)  and  Select  training  exp  4  ...unpause 

On  Participant  GDIS  computer,  select  join  server,  select  server,  enter  password  "imroc",  and 
select  civilian  (blue). 

C2Node  and  SimOCU 

1.  (From  the  desktop)  open  a  Linux  Terminal  window  (right  click  on  desktop). 

2.  (From  the  terminal  prompt)  type: 

for  SIMOCU:  cd  /irl/projects/imroc/exp4/oav/bin 
simocu 

for  C2node:  cd  /irl/projects/imroc/exp4/c2node/bin 

c2node 

SIMOCU: 

File — open — exp4_TR  AININ  G_FIN  ALhp .  opf 
Load  mission — Perimeter.ovm 

C2NODE 
Select  exp4.cnc 

1.  Experimenter  should  read  purpose  of  study  to  participant: 

“This  is  a  joint  project  between  IST/UCF  and  the  Army  Research  Institute.  The  purpose  is  to  test 
an  integrated  system  for  training  operators  of  micro-unmanned  aerial  vehicle  (MAV)  operators 
and  their  commanders.  The  current  effort  will  serve  as  a  test  phase  for  the  technology  and  as  a 
developmental  effort  for  training.  For  the  current  study,  you  will  be  asked  to  operate  two 
computer-based  simulations:  a  command  and  control  station  called  the  C2Node  {point  to 
C2Node}  and  a  first  person  shooter  computer-based  simulation  called  GDIS  {point  to  GDIS}. 
First  you  will  train  and  perform  a  series  of  tasks  within  the  training  environment.  To  ensure  you 
can  navigate  within  the  simulation.  If  for  any  reason  you  do  not  perform  the  tasks  in  3  attempts, 
you  will  be  released  from  the  study  with  credit  for  your  time  to  that  point.  This  is  simply  to 
ensure  that  you  will  be  able  to  operate  within  a  simulated  environment  and  is  no  reflection  on 
your  ability  in  any  other  way.  You  will  then  be  asked  to  perform  a  series  of  4  ‘missions’  where 
you  must  identify  targets  and  positions  of  targets  when  asked.  You  will  also  be  given  a  few 
surveys  both  before  and  after  your  simulation  ‘missions’.  You  should  take  part  in  this  study  only 
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because  you  want  to  and  there  is  no  penalty  for  not  taking  part.  You  have  the  right  to  stop  at  any 
time;  just  tell  the  researcher  or  a  member  of  the  research  team  that  you  want  to  stop  and  you  will 
be  compensated  for  the  amount  of  time  you  actually  participated.  Any  Questions?  Would  you 
like  to  participate?” 


If  participants  agree  ask  them  to  sign: 
Informed  Consent 


2.  Next,  participants  will  complete  paper-based  PowerPoint  Based  training  for  C2Node  and 
GDIS.  Experimenter  should  give  them  the  paper  based  packet  (Part  1).  Tell  the  participant  to 
read  through  the  training  and  ask  questions  at  any  time: 

Part  1:  First  14  Slides.  Let  them  read  and  then  talk  them  show  them  the  keyboard,  mouse,  and 
talk  through  the  training  map.  IDENTIFY  ALL  ICONS  ON  MAP,  INCLUDING  WHERE 
THEY  ARE  STARTING.  Show  them  how  to  rotate  map,  etc.  GIVE  THEM  TRAINING  FORM 
and  show  them  where  to  write  everything. 

SHOW  them  the  doctor  and  others  on  map. 

•  Periodically,  you  will  also  be  asked  to  identify  the  position  of  the  town  Doctor  that  is 
assisting  within  the  area 

•  When  asked,  you  should  pull  up  your  map  (O)  and  find  the  Doctor 

•  Then  consult  your  paper  map  to  determine  what  building  the  doctor  is  near. 

Tell  them  they  will  report  the  position  using  the  text  feature  in  the  C2Node,  which  will  be 
covered  in  the  2nd  part  of  the  training  session. 

Give  them  the  following  instructions: 

“Now  you  will  practice  and  perform  3  tasks  to  ensure  that  you  will  be  able  to  navigate  through 
the  game  and  perform.  First  spend  a  few  minutes  practice  using  the  keys  and  mouse  as  outlined 
in  your  training  to  navigate.” 

When  participant  seems  to  be  able  to  navigate,  give  them  the  following  instructions  (no  more 
than  5  minutes): 

“Now  using  this  map,  travel  to  the  building  labeled  1 1  on  your  map  and  practice  opening  the 
doors  and  exploring  the  building.  Please  try  to  go  through  every  room.  Make  sure  you  find  the 
stairs  to  the  second  floor,  which  are  located  on  the  outside  of  the  building.” 
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Let  participants  go  to  building.  Help  them  get  there  if  they  are  having  trouble.  Make  sure  they  go 
in  every  room  and  find  the  stairs  to  the  upstairs.  Once  they  have  explored  the  building,  give  them 
the  following  instructions: 

“Now  using  the  map,  go  to  the  building  circled  in  blue  on  your  map  and  see  who  (Soldier,  medic, 
refugee)  is  in  the  building.  Again,  make  sure  you  go  through  every  room.  Within  this  building  is 
a  tower,  you  can  climb  it  if  you  would  like,  but  it  is  not  necessary  to  search  there.  Remember  to 
write  down  the  building  number  and  how  many  Soldiers,  refugees,  and  medics  you  find  in  the 
building  on  the  form  I  gave  you.” 

Help  the  participants  get  to  the  building  circled  in  blue  (33).  Give  them  hints  along  the  way. 

“Now  using  the  map,  go  to  the  building  circled  in  red  on  your  map  and  see  who  (Soldier,  medic, 
refugee)  is  in  the  building.  Again,  make  sure  you  go  through  every  room.  Remember  to  write 
down  the  building  number  and  how  many  Soldiers,  refugees,  and  medics  you  find  in  the  building 
on  the  form  I  gave  you.” 

Let  participants  go  to  building  on  their  own.  If  they  go  to  the  wrong  building,  tell  them  they  are 
in  the  wrong  building  and  to  try  again.  At  this  point,  it  is  ok  to  give  them  a  hint  to  as  to  where 
they  went  wrong. 

“Now  using  the  map,  go  to  the  building  circled  in  green  on  your  map  and  see  who  (Soldier, 
medic,  refugee)  is  in  the  building.  Again,  make  sure  you  go  through  every  room.  Remember  to 
write  down  the  building  number  and  how  many  Soldiers,  refugees,  and  medics  you  find  in  the 
building  on  the  form  I  gave  you.” 


If  they  go  to  the  wrong  building  again,  tell  them  they  are  in  the  wrong  building  but  do  not  give 
them  a  hint.  If  they  go  to  the  wrong  building  a  third  time,  thank  them  for  their  participation  and 
inform  them  they  will  get  partial  credit.  If  they  go  to  the  correct  building,  make  sure  they  go  in 
every  room  and  find  the  stairs  to  the  upstairs.  Once  they  have  explored  the  building,  ask  them 
who  and  how  many  people  were  in  the  building.  If  they  answer  wrong,  explain  to  them  what 
went  wrong.  Once  this  exercise  is  complete,  give  them  the  following  instructions: 

“Now  you  need  to  use  both  your  paper  map  and  the  interactive  game  map  to  tell  me  what 
building  Ted  is  near.” 

“Which  building  is  Phil  near?” 

“Which  building  is  the  doctor  near?” 

After  they  answer  these  questions,  ASK  THEM  TO  RETURN  TO  THE  START  POINT. 
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If  participant  gets  all  3  correct,  tell  them  to  return  to  training  to  receive  part  2.  Experimenter 
should  give  them  the  next  paper  based  packet  (Part  2).  Tell  the  participant  to  read  through  the 
training  and  ask  questions  at  any  time: 

Part  2:  rest  of  the  slides.  Let  them  read  and  remind  them  to  ask  questions  along  the  way. 

Part  2  consists  of  Slides  20  through  42. 

Select  take  off  for  MAV  and  automation  to  follow  path 

Once  the  participants  have  been  through  training,  give  them  the  following  instructions: 

“Now  you  will  take  a  few  minutes  to  make  sure  you  know  how  to  use  the  C2Node  or  command 
and  control  unit.  Please  respond  to  the  requests  you  receive  in  the  text  window.” 

Experimenter  will  show  participant  relevant  cues  in  C2Node.  MAKE  SURE  TO  SHOW  WHAT 
THEY  ARE,  THE  MAV  IS,  AND  THE  VEHICLE  IS.  Also,  how  to  use  text  and  how  to  find  grid 
numbers. 

Experimenter  should  send  picture  of  ***  to  participant  using  the  SimOCU  and  text  them  the 
phrase:  “Is  there  a  hummer  in  picture  1?” 

Participant  should  text  back  “***”.  if  incorrect,  tell  participant  what  is  in  the  picture. 

Experimenter  should  send  picture  of  ***  to  participant  using  the  SimOCU  and  text  them  the 
phrase:  “Is  there  a  hummer  in  picture  2  ?” 

Participant  should  text  back  “***”.  if  incorrect,  tell  participant  what  is  in  the  picture. 

Experimenter  should  send  picture  of  ***  to  participant  using  the  SimOCU  and  text  them  the 
phrase:  “Is  there  a  hummer  in  picture  3  ?” 

Participant  should  text  back  “***”.  if  incorrect,  tell  participant  what  is  in  the  picture. 


Experimenter  text  participant  the  phrase:  “Please  identify  the  position  of  the  MAV  using  the  first 
4  grid  coordinates.” 

Experimenter  text  participant  the  phrase:  “Please  identify  the  position  of  the  Doctor.” 

Participant  should  text  back  a  4  digit  number  for  the  MAV  and  the  2  digit  number  of  the  building 
for  the  Doctor.  If  incorrect,  text  and  ask  participant  to  report  the  position  now  of  each  up  to  3 
times. 

***If  participant  cannot  complete  these  requests,  please  thank  them  for  coming  and  inform  them 
they  will  receive  credit  for  the  time  they  have  participated. 
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Inform  participant  that  they  have  completed  training  and  can  now  proceed  to  the  performance 
missions.  First  they  must  complete  the  following  surveys. 


Following  training,  participants  will  complete  while  experimenter  is  loading  mission  maps: 

1.  Computer  based  NASA  TLX 

2.  Demographics  Questionnaire 

3.  Cube  Comparison  Test 

Experimenter  should  continue  to  the  appropriate  protocol  (Condition  1-4)  and  follow  directions. 
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